More micro benchmarks #1158

BrennanConroy · 2017-11-27T18:47:34Z

             Method |  Input | HubProtocol |        Mean |     Error |    StdDev |        Op/s |  Gen 0 | Allocated |
------------------- |------- |------------ |------------:|----------:|----------:|------------:|-------:|----------:|
  ReadSingleMessage |  Large |        Json | 18,124.9 ns | 377.49 ns | 565.01 ns |    55,172.8 | 0.1831 |   17321 B |
 WriteSingleMessage |  Large |        Json | 11,894.0 ns | 289.01 ns | 432.57 ns |    84,076.0 | 0.1968 |   18248 B |
  ReadSingleMessage |  Large |     MsgPack |  4,630.1 ns | 167.94 ns | 251.36 ns |   215,979.3 | 0.0603 |    5920 B |
 WriteSingleMessage |  Large |     MsgPack |  5,752.0 ns | 485.18 ns | 726.19 ns |   173,852.7 | 0.1053 |    9656 B |
  ReadSingleMessage | Medium |        Json | 12,367.9 ns | 519.64 ns | 777.77 ns |    80,854.2 | 0.0870 |    9504 B |
 WriteSingleMessage | Medium |        Json |  5,507.0 ns | 265.29 ns | 397.08 ns |   181,587.1 | 0.0763 |    7272 B |
  ReadSingleMessage | Medium |     MsgPack |  2,362.9 ns | 116.92 ns | 175.00 ns |   423,207.7 | 0.0076 |     992 B |
 WriteSingleMessage | Medium |     MsgPack |  2,076.2 ns | 109.59 ns | 164.04 ns |   481,655.4 | 0.0114 |    1136 B |
  ReadSingleMessage |  Small |        Json |  4,654.5 ns | 207.23 ns | 310.17 ns |   214,845.5 | 0.0740 |    6888 B |
 WriteSingleMessage |  Small |        Json |  2,055.4 ns |  78.02 ns | 116.77 ns |   486,530.6 | 0.0687 |    6464 B |
  ReadSingleMessage |  Small |     MsgPack |    441.5 ns |  21.56 ns |  32.27 ns | 2,265,137.4 | 0.0048 |     448 B |
 WriteSingleMessage |  Small |     MsgPack |    553.9 ns |  13.63 ns |  20.39 ns | 1,805,414.2 | 0.0086 |     832 B |

BrennanConroy · 2017-11-28T17:12:47Z

Added another micro benchmark for broadcasting to many clients using DefaultHubLifetimeManager
Also, benchmarks now do a validation run during ./build.cmd to make sure we didn't break them.

         Method | Connections |       Mean |      Error |     StdDev |      Op/s |  Gen 0 | Allocated |
--------------- |------------ |-----------:|-----------:|-----------:|----------:|-------:|----------:|
 InvokeAsyncAll |           1 |   1.367 us |  0.1058 us |  0.1583 us | 731,520.7 | 0.0002 |     502 B |
 InvokeAsyncAll |          10 |   3.205 us |  0.3724 us |  0.5574 us | 311,993.2 |      - |    1064 B |
 InvokeAsyncAll |        1000 | 332.336 us | 32.7849 us | 49.0709 us |   3,009.0 |      - |   62865 B |

analogrelay · 2017-11-28T17:57:11Z

Hmm, wonder if we should put the connection count in as the Operation count for each of those iterations. The "Op/s" value is kinda distractingly unrelated since there are inherently more operations performed in each InvokeAsync when there are more connections. I'm not sure you can bind a parameter to the operation count though since it's an attribute.

BrennanConroy · 2017-11-28T18:01:40Z

We should be aware of what Op/s means here, we can add a comment to the benchmark at a minimum
"Op/s is measuring the number of broadcasts the server can perform with the specified number of clients"

analogrelay

Looks good but I already want more ;P

analogrelay · 2017-11-28T18:01:52Z

benchmarks/Microsoft.AspNetCore.SignalR.Microbenchmarks/HubProtocolBenchmark.cs

+        private TestBinder _binder;
+        private HubMessage _hubMessage;
+
+        [Params(Message.Small, Message.Medium, Message.Large)]


Use descriptive names rather than Small/Medium/Large. Also, there are kinda two axes here: Number of payload values, size of payload values. We should see if we can cross-product them a little (e.g. a test that serializes a single 4K string, and a test that serializes 4K 1 byte strings). I think aiming for 2-3 "expected scenarios" (e.g. 3-4 small values, 8-10 small values, 1 very large value) and then a couple "extreme" scenarios (lots of small values, one extraordinarily large value) would be good. Assuming that doesn't balloon the run time too much :).

That would balloon it like crazy, I think it already takes 13 minutes for this benchmark

analogrelay · 2017-11-28T18:02:12Z

benchmarks/Microsoft.AspNetCore.SignalR.Microbenchmarks/HubProtocolBenchmark.cs

+            switch (Input)
+            {
+                case Message.Small:
+                    _hubMessage = new CancelInvocationMessage("123");


Let's use the same message type for all of these tests. If that makes "Small" bigger, so be it.

analogrelay · 2017-11-28T18:09:12Z

What we're most interested in is how well it scales. Is it linear per connection (ideal) or is there some multiplier effect as more connections arrive. As long we understand clearly how to find that data, I don't mind. Ideally, Op/s would be the value we expect to scale with connections, but it may be difficult to make the infrastructure support that.

BrennanConroy · 2017-11-28T18:12:50Z

So ignoring 1 connection for now, 10 connections was ~300k Op/s, 1000 connections was ~3k Op/s, not shown but done during testing 100 connections was ~30k Op/s
Looks fairly linear

analogrelay · 2017-11-28T18:22:08Z

Nice

analogrelay

Reach for the moon @BrennanConroy ! Bigger payload! Otherwise good to go (and if that makes the benchmark super slow, just make it a power of two and merge as-is)

analogrelay · 2017-11-28T23:02:02Z

benchmarks/Microsoft.AspNetCore.SignalR.Microbenchmarks/HubProtocolBenchmark.cs

+                    _hubMessage = new InvocationMessage("123", true, "Target", null, 1, "string", 2.0f, true, (byte)9, new byte[] { 5, 4, 3, 2, 1 }, 'c', 123456789101112L);
+                    break;
+                case Message.LargeArguments:
+                    _hubMessage = new InvocationMessage("123", true, "Target", null, new string('F', 1234), new byte[1000]);


larger Larger

LARGER

But seriously, let's try 10 KB (10240) and see how it performs, if the benchmark is super slow, then scale it back. Also use a power of two, not a power of ten, jeez.

Psh power of two is too clean, computers like those, we don't want to be nice to the computer

BrennanConroy added 6 commits November 27, 2017 10:24

begin bench

f6f04a4

more bench

df147fb

progress

eeb2d3f

clean

09fc230

private

ae933a9

updates

05781cd

BrennanConroy changed the title ~~HubProtocol micro benchmarks~~ More micro benchmarks Nov 28, 2017

cleanup

8e8417a

analogrelay reviewed Nov 28, 2017

View reviewed changes

BrennanConroy mentioned this pull request Nov 28, 2017

Performance Investigation #1164

Closed

6 tasks

better?

971ae58

analogrelay approved these changes Nov 28, 2017

View reviewed changes

bigger

78b23f8

BrennanConroy merged commit 531c7cf into dev Nov 28, 2017

BrennanConroy deleted the brecon/bench branch November 28, 2017 23:40

More micro benchmarks #1158

More micro benchmarks #1158

Uh oh!

Conversation

BrennanConroy commented Nov 27, 2017

Uh oh!

BrennanConroy commented Nov 28, 2017

Uh oh!

analogrelay commented Nov 28, 2017

Uh oh!

BrennanConroy commented Nov 28, 2017

Uh oh!

analogrelay left a comment

Choose a reason for hiding this comment

Uh oh!

analogrelay Nov 28, 2017

Choose a reason for hiding this comment

Uh oh!

BrennanConroy Nov 28, 2017

Choose a reason for hiding this comment

Uh oh!

analogrelay Nov 28, 2017

Choose a reason for hiding this comment

Uh oh!

analogrelay commented Nov 28, 2017

Uh oh!

BrennanConroy commented Nov 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

analogrelay commented Nov 28, 2017

Uh oh!

analogrelay left a comment

Choose a reason for hiding this comment

Uh oh!

analogrelay Nov 28, 2017

Choose a reason for hiding this comment

LARGER

Uh oh!

BrennanConroy Nov 28, 2017

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BrennanConroy commented Nov 28, 2017 •

edited

Loading